Image storage and retrieval based on eye movements

ABSTRACT

Various embodiments are generally directed to creating and using an index based on eye movements of the human eye to store and retrieve images in an image database. An apparatus comprises a processor circuit and a storage communicatively coupled to the processor circuit and storing instructions operative on the processor circuit to receive a first eye movement data associated with a first image provided by the apparatus from an image database stored in the storage; determine a first identity of a first object at a first focus region in the first image indicated by the first eye movement data; search the image database for an image depicting the first object; and provide a second image depicting the first object from the image database. Other embodiments are described and claimed herein.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, claims the benefit of andpriority to, previously filed U.S. patent application Ser. No.13/629,706, entitled “IMAGE STORAGE AND RETRIEVAL BASED ON EYEMOVEMENTS”, filed Sep. 28, 2012, the subject matter of which isincorporated herein by reference in its entirety.

BACKGROUND

The use of text-based search engines to locate information, includingimages, stored within a single computing device or across a great manycomputing devices accessible via a network (e.g., the Internet) iscommonplace. Unfortunately, while text-based searching has proven quiteeffective in finding items of text (e.g., literature, lyrics, etc.), aswell as non-text items that are somehow linked to descriptive text, itoften proves of limited value in locating an image (whether a stillimage or an image of a video) that depicts a particular sought-for item.

A longstanding approach to enabling a search for an image depicting aparticular item has been the manual tagging of images, by people, withdescriptive text. Unfortunately, given that images are captured by allkinds of people all over the world with varying opinions of whatconstitutes an effective description, the effectiveness of suchdescriptive text varies widely. Also, there is a tendency among manypeople to describe only objects in their captured images that areimportant to them, thereby failing to describe other objects in theircaptured images that may be of interest to other people. Further, thereare many people who capture images, but never actually tag them with anydescriptive text, at all, sometimes simply as a result of finding itdifficult to use text-labeling features of image-handling devices tocreate such textual tags. As a result, there are a great many imagesthat are not tagged with any searchable text description, whatsoever.

One approach to resolving insufficient or missing descriptive text formany images has been to employ people to review large numbers of imagesand manually create or edit textual tags. Unfortunately, such anapproach is time consuming and quickly becomes cost-prohibitive. Anotherpast approach attempts to resolve these problems by automating thetextual tagging of images. Specifically, computing devices have beenused to scan images, employ various visual recognition algorithms toidentify everything that is depicted, and tag those images withautomatically-generated text listing the objects identified.Unfortunately, such an approach tends to require considerable computingresources, and can misidentify or fail to identify objects in thoseimages. Further, neither of these approaches can discern what object(s)in those images were of interest to the people who captured them, or therelationships between objects. It is with respect to these and otherconsiderations that the techniques described herein are needed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a first embodiment of interaction among computingdevices.

FIG. 2 illustrates aspects of processing performed in the embodiment ofFIG. 1.

FIG. 3 illustrates a portion of the embodiment of FIG. 1.

FIG. 4 illustrates a portion of the embodiment of FIG. 1.

FIG. 5 illustrates a portion of the embodiment of FIG. 1.

FIG. 6 illustrates a first embodiment of interaction among computingdevices.

FIG. 7 illustrates an embodiment of a first logic flow.

FIG. 8 illustrates an embodiment of a second logic flow.

FIG. 9 illustrates an embodiment of a third logic flow.

FIG. 10 illustrates an embodiment of a fourth logic flow.

FIG. 11 illustrates an embodiment of a processing architecture.

DETAILED DESCRIPTION

Various embodiments are generally directed to creating and using anindex based on detected aspects of eye movements of the human eye tostore and retrieve images in an image database. More specifically, aneye tracker is employed to monitor the movements of a person's eye whileviewing an image to reveal focus regions (areas of an image at which agreater proportion of fixations occur over time than other areas) andsaccades (movements of their eye between fixations). Focus regions areinferred to be areas of an image where objects of interest to thatperson exist. Higher proportions of saccades occurring between somefocus regions versus others are inferred to indicate relationships thatmay exist among objects depicted in the focus regions between which thesaccades occur, and/or are inferred to indicate spatial relationshipsbetween objects depicted in a focus region and fixations surroundingthem between which the saccades occur.

Focus regions are employed as indications of areas of an image to whichvisual recognition and/or other automated algorithms may be directed toidentify the objects of the image found to be at those focus regions toautomatically generate tags for use in indexing a database of images.Also incorporated into the index are indications of possiblerelationships inferred as exiting between objects within focus regionsbetween which a high proportion of saccades (or series of saccades) arefound to occur. Further, spatial relationships between objects at focusregions and fixations between which a high proportion of saccades arealso found to occur are also incorporated into the index.

Tracking of eye movements may occur as an image is captured by trackingthe eye movements of an eye of the person operating a capture device tocapture the image. In so doing, indications of what object(s) in thatimage are of interest to that person and relationships concerning thoseobject(s) inferred as perceived by that person are incorporated into theindex. Later, tracking of eye movements may occur as that image isviewed by other people during searches of images by tracking the eyemovements of each such other person as they view the same image. In sodoing, search results of images presented to each such person may bemore quickly narrowed to images depicting objects similar to or relatedto objects of interest to them. Also in so doing, indications of whatobject(s) in that image are of interest to each of them andrelationships concerning those object(s) inferred as perceived by themare also incorporated into the index.

It is expected that each person looking at a given image willdemonstrate their own unique eye movements stemming from and leading totheir unique focus regions and unique relationships associated withobjects identified at those focus regions. Thus, over time, the indexingfor an image originally having only information gleaned from eyemovements of the person that captured it will eventually incorporateinformation gleaned from eye movements of many people, therebyuncovering more focus regions, more objects to be identified, and morerelationships involving those objects. As the index is increasinglyaugmented with such information, its effectiveness in supportingsearches for images depicting objects of interest is increasinglyimproved.

In one embodiment, for example, an apparatus comprises a processorcircuit and a storage communicatively coupled to the processor circuitand storing instructions operative on the processor circuit to receive afirst eye movement data associated with a first image provided by theapparatus from an image database stored in the storage; determine afirst identity of a first object at a first focus region in the firstimage indicated by the first eye movement data; search the imagedatabase for an image depicting the first object; and provide a secondimage depicting the first object from the image database. Otherembodiments are described and claimed herein.

With general reference to notations and nomenclature used herein,portions of the detailed description which follows may be presented interms of program procedures executed on a computer or network ofcomputers. These procedural descriptions and representations are used bythose skilled in the art to most effectively convey the substance oftheir work to others skilled in the art. A procedure is here, andgenerally, conceived to be a self-consistent sequence of operationsleading to a desired result. These operations are those requiringphysical manipulations of physical quantities. Usually, though notnecessarily, these quantities take the form of electrical, magnetic oroptical signals capable of being stored, transferred, combined,compared, and otherwise manipulated. It proves convenient at times,principally for reasons of common usage, to refer to these signals asbits, values, elements, symbols, characters, terms, numbers, or thelike. It should be noted, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to those quantities.

Further, these manipulations are often referred to in terms, such asadding or comparing, which are commonly associated with mentaloperations performed by a human operator. However, no such capability ofa human operator is necessary, or desirable in most cases, in any of theoperations described herein that form part of one or more embodiments.Rather, these operations are machine operations. Useful machines forperforming operations of various embodiments include general purposedigital computers as selectively activated or configured by a computerprogram stored within that is written in accordance with the teachingsherein, and/or include apparatus specially constructed for the requiredpurpose. Various embodiments also relate to apparatus or systems forperforming these operations. These apparatus may be speciallyconstructed for the required purpose or may comprise a general purposecomputer. The required structure for a variety of these machines willappear from the description given.

Reference is now made to the drawings, wherein like reference numeralsare used to refer to like elements throughout. In the followingdescription, for purposes of explanation, numerous specific details areset forth in order to provide a thorough understanding thereof. It maybe evident, however, that the novel embodiments can be practiced withoutthese specific details. In other instances, well known structures anddevices are shown in block diagram form in order to facilitate adescription thereof. The intention is to cover all modifications,equivalents, and alternatives within the scope of the claims.

FIG. 1 illustrates a block diagram of an image storage and retrievalsystem 1000 comprising one or more of a capture device 100 operable by aperson capturing images, a server 300 storing images, and a viewingdevice 500 operable by a person searching for and viewing images. Eachof the computing devices 100, 300 and 500 may be any of a variety oftypes of computing device, including without limitation, a desktopcomputer system, a data entry terminal, a laptop computer, a netbookcomputer, a tablet computer, a ultrabook computer, a handheld personaldata assistant, a smartphone, a digital camera, a body-worn computingdevice incorporated into clothing, a computing device integrated into avehicle, a server, a cluster of servers, a server farm, etc.

As depicted, the computing devices 100, 300 and 500 exchange signalsconveying at least image data and data used in searching for imagesthrough a network 999, although one or more of these computing devicesmay exchange other data entirely unrelated to images. In variousembodiments, the network 999 may be a single network possibly limited toextending within a single building or other relatively limited area, acombination of connected networks possibly extending a considerabledistance, and/or may include the Internet. Thus, the network 999 may bebased on any of a variety (or combination) of communicationstechnologies by which signals may be exchanged, including withoutlimitation, wired technologies employing electrically and/or opticallyconductive cabling, and wireless technologies employing infrared, radiofrequency or other forms of wireless transmission.

In various embodiments, and as will be explained in greater detail, thecapture device 100 is operated by a person to capture an image and anytextual tags that may be manually entered by that person. Also, duringcapture of the image, eye movements of an eye of that person areautomatically captured, along with any tags concerning the image thatare automatically generated by the capture device 100 (e.g., GPScoordinates, camera compass direction, etc.). Subsequently, the capturedevice 100 transmits the captured image and associated informationconcerning eye movements and/or tags to the server 300 via the network999 (or through another mechanism, possibly through another computingdevice coupled to the network 999) to be added to a database maintainedby the server 300.

Upon receipt of the captured image and associated eye and/or taginformation from the capture device 100, the server 300 stores thecaptured image in an image database, and adds any such received taginformation to an index used by server 300 in locating images in theimage database. The server 300 examines the eye movements of theoperator of the capture device 100 to determine locations of fixationsand the saccades occurring therebetween. From the fixations andsaccades, the server 300 further determines focus regions of the imagein which clusters of higher proportions of fixations occurred, anddetermines where higher proportions of saccades occurred between focusregions. The server 300 then employs various visual recognition and/orfeature extraction algorithms to identify objects present in thecaptured image at the focus regions, and augments the index withindications of those identified objects. The server 300 further augmentsthe index with indications of relationships inferred as possiblyexisting between objects located at focus regions between which higherproportions of saccades are found to occur (either single saccadesdirectly between those focus regions, or series of saccades throughfixations not associated with either of those focus regions), andspatial relationships of objects located at focus regions and fixationswhere higher proportions of saccades are found to occur between thosefocus regions and those fixations.

It should be noted that directing the use of visual recognitionalgorithms to identify objects or other depicted at the locations offocus regions in an image, instead of employing visual recognitionalgorithms to identify all objects depicted throughout an image, makesadvantageous use of the indications provided by focus regions of wherethere are objects in the image that are deemed to be of sufficientimportance as to be useful to identify versus other objects in theimage. Visual recognition algorithms typically consume considerablecomputational resources, and such targeting of the use of visualrecognition algorithms towards focus regions makes more efficient use ofthose resources. Further, the use of visual recognition algorithms toidentify all objects in an image can generate a far greater quantity ofdata for each image, requiring considerably more in the way of storageresources, and such targeting of the use of visual recognitionalgorithms towards focus regions avoids generating considerablequantities of unnecessary data requiring storage. Still, it may be thatthe server 300 employs one or more recognition algorithms optimized toanalyze broader characteristics of the captured image captured by thecapture device 100 to broadly categorize it (e.g., as outdoor scenery,as indoor scenery, as artificially created scenery created through paintor computer-generated animation, etc.), and then augment the index toinclude such additional tag information.

At a later time, the viewing device 500 is operated by another person(or possibly the same person who operated the capture device 100 tocapture images) to search for an image depicting a particular object ofinterest to them. That person may initially begin their search in any ofa variety of ways, including and not limited to, entering or in someother way selecting text to search for images depicting what theydescribe in that entered text. Such operation of the viewing device 500results in the viewing device 500 transmitting the entered text to theserver 300 via the network 999 as search terms to be used in searchingfor an image. In response, the server 300 searches its database,employing its index to retrieve one or more initial images tagged withtext information matching and/or related to the text received from theviewing device 500. The server 300 then transmits the initial multitudeof retrieved images to the viewing device 500 via the network 999.

Upon receipt of these one or more initial images retrieved from theserver 300, the viewing device 500 visually presents the initialimage(s) on a display of the viewing device 500, allowing the personoperating the viewing device 500 to view them. As this person viewsthese image(s), eye movements of at least one of their eyes arecaptured, enabling at least some of their fixations and saccades on oneor more images they view to be identified. From these, focus regions andsaccades related to those focus regions are identified. The identifyingof fixations, saccades, focus regions, and/or saccades related to thosefocus regions, at least partly, may be at least partly performed by theviewing device or may be performed entirely by the server 300. However,the focus regions and saccades related thereto are identified, theserver 300 employs this information to retrieve one or more additionalimages from the database that depict objects matching or in some wayrelated to the objects depicted at those focus regions, and thentransmits the one or more additional images to the viewing device 500.Upon receiving the additional image(s), the viewing device 500 displaysthe additional image(s), either in addition to or in lieu of the initialone or more images. Thus, eye movements are employed to refine thesearch for an image, and this refinement may be repeated by againcapturing eye movements as the viewing device 500 is operated to viewthe additional image(s) to yet further refine searching.

The server 300 may also employ the information concerning focus regionsand saccades related thereto to augment the index where at least some ofthose focus regions are found to occur in images at locations wherefocus regions were not previously identified. If one or more of thesefocus regions are not already included in the index, then the server 300employs various visual recognition algorithms to identify objects atthose newly identified focus regions, and augments the index withindications of those newly identified objects. Thus, the viewing ofimages of the database maintained by the server 300, over time and bynumerous people using such devices as the viewing device 500, results inthe index being improved over time so that desired images already in thedatabase may be found more readily by others in future searches.

This ability to augment the index enables the index to adapt over timeas cultural and societal changes bring about changes to what objectsthat may be depicted in images are generally of interest to people. Thisalso addresses the issue of an image captured with the capture device100 initially having information concerning fixations, saccades, focusregions, and/or other aspects of eye movements only of the person whooperated the capture device 100 to capture it. As a result of thisability to augment the index, the eye movements of an eye of the personoperating the capture device 100 do not indefinitely remain the onlyinfluence on the effectiveness with which the image may be found infuture searches.

In various embodiments, the capture device 100 comprises a storage 160storing a control routine 140 and an image data 138, a processor circuit150, controls 120, a display 180, a camera 110, an eye tracker 181, andan interface 190 coupling the capture device 100 to the network 999. Theimage data 138 comprises eye movement data 131 and tag data 132. Inexecuting a sequence of instructions of at least the control routine140, the processor circuit 150 is caused to monitor the controls 120 toenable an operator of the capture device 100 to operate the controls 120to signal the processor circuit 150 with a command to operate the camera110 to capture an image of whatever is in the view of the camera 110.

It is envisioned that, being a computing device configured to serve as adigital camera device (e.g., a smartphone with a built-in camera, adigital camera, etc.), the operator views the display 180 to see what isin the view of the camera 110 as they operate the controls 120 to signalthe capture device 100 to capture an image. In other words, it isenvisioned that the display 180 serves as an electronic equivalent of acamera eyepiece. Therefore, the eye tracker 181 is positioned relativeto the display so as to be able to track eye movements of at least oneeye of the operator as the operator looks at the display 180 to aim thecamera 110 at whatever they choose to capture an image of. The processorcircuit 150 is caused, in response to operation of the controls 120 bythe operator, to track those eye movements, and identify the fixationsand saccades that occur in what ultimately becomes a captured imagedisplayed on the display 180. Alternatively or additionally, it isenvisioned that, after having captured the image, the operator may againview the captured image on the display 180, thereby potentiallyproviding another opportunity to track their eye movements as they viewthe captured image.

Studies of the manner in which the human brain (in particular, thevisual cortex) performs in order to view an image reveal that, unlike atypical computing device employing a raster-scan camera, the human braindoes not perform the equivalent of single-pass detailed “scan” of anentire image as a single unitary piece of information. Instead, thehuman brain initiates movement of the eyes about the image in a sequenceof fixations and saccades. Fixations are observations of a certain pointin the visual field (with a typical duration of 200 ms to 300 ms)leading to an accurately processed scanning of approximately 2° of thevisual field, and saccades are eye movements relocating the point offixation (with a typical duration of 40 ms, depending on the saccadeamplitude) during which there is suppression of sight such that thesaccades are not actually perceived as having occurred. In this way, thehuman brain builds up a mental picture of what is shown in the image ina somewhat piecemeal fashion. As time passes, what objects in an imageare more of interest than others to a particular person becomesincreasingly clear as a greater proportion of their fixations begin toaccumulate in a clustered fashion about one or more focus regions atwhich objects of interest are located within the image. Further,saccades might occur in increasingly greater proportion between specificfocus regions, giving rise to the possibility of there being arelationship between one or more objects at one focus region and one ormore objects at another focus region. It should be noted that suchsaccades occurring in high proportion between two particular focusregions may not be single saccades occurring directly between them, butmay include series of saccades separated by fixations not associatedwith any focus region. Still further, as time passes, saccades occurringbetween a focus region and one or more fixations not associated with afocus region may also suggest a spatial relationship between that focusregion and either those fixations or other focus regions.

FIG. 2 depicts an example image 880 captured by the camera 110 of thecapture device 100. Highlighted with dashed circles is an example of apair of focus regions, and highlighted with a short segment extendingbetween the circles is the occurrence of a high proportion of saccades881 occurring between those two focus regions. As depicted, one of thefocus regions is centered on an image of a top of a fir tree, andanother is centered on an image of a cloud. Each fixation typicallycovers an area of an image that corresponds to about two degrees ofmovement around the center of the fixation. As a result, the area of animage that is covered by a single fixation varies in its size withdistance from a human eye. A focus region comprises numerous fixationsthat are closely spaced, but many of which do not completely overlap,and thus, the area of an image that is covered by a focus region tendsto be somewhat wider than that of a single fixation.

As also depicted in FIG. 2, as the image 880 is captured by the camera110, it is stored as the image data 138, which also comprises an eyemovement data 131 and a tag data 132. Typically, a person operating acamera will delay operating a control to cause capturing of an image tooccur until that person clearly sees the image, themselves, well enoughto refine their aiming of the camera. This typically provides anopportunity for the eye tracker 181 to track eye movements of at leastone of the person's eyes to capture their eye movements prior to theiroperating the controls 120 to capture an image. The processor circuit150 is caused to respond to such operation of the controls 120 bycapturing indications of the eye movement observed via the eye tracker181 to have occurred immediately preceding the time at which thecontrols 120 were so operated. Further, if that person then subsequentlyviews the now captured image on the display 180, the eye tracker 181 mayagain be operated by the processor circuit 150 to capture indications oftheir eye movements.

Although the eye tracker 181 may be based on any of a variety oftechnologies, it is envisioned that at least some implementations of theeye tracker 181 detect the direction in which the gaze of at least oneeye at a recurring interval (perhaps coincident with or a multiple ofthe refresh rate of the display 180). With each such interval, a gazepoint (e.g., a point on the display 180 at which that eye is gazing) isdetermined, and thus, the eye movement data 131 may comprise a series ofsuch gaze points at such a temporal interval. In analyzing these gazepoints, where a series of gaze points indicates a gaze remaining atsubstantially the same location on the display 180 (possibly for atleast a selected period of time used to identify fixations), a fixationmay be determined to have occurred at that point. Where a series of gazepoints show the location at which the eye gazes on the display as movingbetween adjacent gaze points, a saccade may be determined to haveoccurred through those gaze points. It may be that the processor 150 iscaused to analyze these gaze points and determine where fixations andsaccades have occurred, and then store those as the eye movement data131, possibly in lieu of storing gaze points as the eye movement data131. Alternatively, the eye movement data 131 may comprise gaze points,and it may be the processor 350 of the server 300 that determine wherefixations and saccades have occurred.

The tag data 132 may comprise one or more of text entered by theoperator via the controls 120 to manually provide a textual descriptionof the image 880, a timestamp possibly including both date and time atwhich the image 880 was captured, and/or GPS coordinates (or otherlocation information) of the place at which the image 880 was captured.Of the text that may be manually entered by the operator, the text maybe a title that the operator gives the image 880, a descriptioncomprising one or more sentences, key words, and/or the name of a filecomprising the image data 138.

Returning to FIG. 1, having captured and stored the image 880 as theimage data 138, along with the eye movement data 131 and the tag data132, the processor circuit 150 is further caused to signal the server300 via the network 999 to convey the image data 138 to the server 300via the network 999 for storage by the server 300. Depending on theaccessibility of the network 999 to the capture device 100 and/or thepreferences of its operator, such signaling of the server 300 may occurrelatively immediately following image capture, or may occur at a latertime of the operator's choosing. Further, it may be that the operatorfirst copies the image data 138 to another computing device in thepossession of the operator (not shown), possibly to allow the operatorto view and evaluate the image 880 on that other computing device beforedeciding to transmit it to the server 300.

In various embodiments, the server 300 comprises a processor circuit350; an interface 390 coupling the server 300 to the network 999; and astorage 360 storing one or more of a control routine 340, the image data138, an image database 330, an eye movement data 531 and a tag data 532.The image database 330 comprises an index 335 and numerous pieces ofimage data 338 a through 338 x that each comprise an image. In executinga sequence of instructions of at least the control routine 340, theprocessor circuit 350 is caused to operate the interface 390 to receivethe aforementioned signal from the capture device 100 conveying theimage data 138 (including the eye movement data 131 and the tag data132) to the server 300.

Upon receipt of the image data 138, the processor circuit 350 is causedto store the image 880 of the image data 138 in the image database 330as one of the image data 338 a through 338 x, and is caused to augmentthe index 335 with the tag data 132. The processor circuit 350 alsoemploys various visual recognition algorithms to identify objectsdepicted in the image 880 at the locations of the focus regions eitherindicated directly by the eye movement data 131 or derived by theprocessor circuit 350 from the eye movement data 131. The processorcircuit 350 then augments the index 335 with indications of thoseidentified objects. The server 300 further augments the index 335 withindications of possible relationships among objects within pairs offocus regions between which high proportions of saccades have occurred.The server 300 yet further augments the index 335 with indications ofpossible spatial relationships of focus regions (or the objectsidentified at focus regions) with one or more fixations between which ahigh proportion of saccades have been found to have occurred. It may bethat the processor circuit 350 is caused to employ one or more visualrecognition algorithms to perform some limited degree of identificationof objects at such fixations. Alternatively or additionally, the server300 augments the index 335 with data indicating the pattern of eyemovements (possibly gaze points) observed by an eye tracker (e.g., theeye tracker 181).

It should be noted that the objects identified as located at focusregions may be any of a wide variety of types of objects. These objectsmay comprise simple lines or shapes or colors located at a focus region.Alternatively or additionally, these objects may comprise identifiedphysical objects, such as without limitation, people, buildings, plants,furniture, vehicles, etc. Still further, these objects may comprisederived histograms of gradients of color(s), directions of detectedlines, aspects of depicted textures, etc.

It should be noted that although the image database 330 is depicted anddiscussed herein as having the index 335 separate and distinct from theimage data 338 a-x, this is but one possible organization of elementswithin a database, as those skilled in the art of database organizationwill readily recognize. For example, as an alternative, the contents ofthe index 335 may be distributed among the image data 338 a-x such that,similar to the image data 138, each of the image data 338 a-x comprisestags, eye movement data indicating fixations and saccades, and/or otherinformation operative to serve as bases for searching for images amongthe image data 338 a-x.

In various embodiments, the viewing device 500 comprises a processorcircuit 550, a display 580, an eye tracker 581, controls 520, aninterface 590 coupling the viewing device 500 to the network 999, and astorage 560 storing a control routine 540 and one or more of the imagedata 338 a-x, eye movement data 531 and tag data 532. In executing asequence of instructions of at least the control routine 540, theprocessor circuit 550 is caused to monitor the controls 520 to enable anoperator of the viewing device 500 to operate the controls 520 to signalthe processor circuit 550 with a command to signal the server 300 tosearch for an image.

The operator may initially begin the search by operating the controls520 to manually enter text or to otherwise select of specify text thatspecifies an object of which the operator desires to find an image. Theprocessor circuit 550 is caused to store that text as the tag data 532,and is then caused to operate the interface 590 to send the tag data 532to the server 300 via the network 999. In response to receiving the tagdata 532, the processor circuit 350 of the server 300 is caused toemploy the index 335 in searching the image database 330. From the imagedatabase 330, the processor circuit 350 retrieves a first subset of theimage data 338 a-x comprising a multitude of images tagged withinformation matching and/or related to the search text of the tag data532. The processor circuit 350 is then caused to operate the interface390 to transmit this first subset of the image data 338 a-x via thenetwork 999 to the viewing device 500.

Upon receipt of this first subset of the image data 338 a-x, theprocessor circuit 550 of the viewing device 500 is caused to display theimages of that first subset on the display 580, allowing the operator ofthe viewing device 500 to view them. As the operator views these images,eye movements of at least one of their eyes are monitored by the eyetracker 581, enabling their fixations and saccades occurring on each ofthe images they view to be determined, and ultimately enabling focusregions and/or where higher proportions of saccades have occurred fromwhich relationships might be inferred. It may that the eye movement data531 comprises little more than gaze points indicating locations on thedisplay 580 at which the monitored eye has been determined to be gazingat a recurring interval. Alternatively or additionally, the processorcircuit 550 may be caused by the control program 540 to determine thelocations of fixations and saccades from the observed movements of themonitored eye, and to store those fixations and saccades as part of theeye movement data 531. Yet again, alternatively or additionally, theprocessor circuit 550 may be caused to derive focus regions and/or wherethere are saccades occurring in higher proportions between focusregions. Regardless of the eye movement data 531 exactly comprises, theprocessor circuit 550 is then caused to transmit the eye movement data531 to the server 300, thereby providing the server with indications ofwhich one(s) of the images of the first subset of the image data 338 a-xare the image(s) in which fixations, saccades, focus regions, etc. haveoccurred that provide an indication of objects of interest to the viewerin those images.

Regardless of the exact nature of the data of which the eye movementdata 531 is composed, if there is an indication in the eye movement data531 of one or more focus regions occurring at locations in those imagesat which focus regions had been previously identified, then theprocessor circuit 350 is caused to employ this information of the eyemovement data 531 to retrieve indications from tag data within the index335 identifying what objects are at those locations. The processorcircuit 350 is then caused to use this information concerning theidentities of those objects at those locations to search for additionalimages within the image database 330 that depict matching or relatedobjects. The processor circuit 350 then operates the interface 390 totransmit a second subset of the image data 338 a-x that comprises thoseadditional images to the viewing device 500. Upon receiving the secondsubset of the image data 338 a-x comprising the additional images, theprocessing device 550 is caused to display those additional images onthe display 580, either in addition to or in lieu of the images of thefirst subset of the image data 338 a-x.

However, if the eye movement data 531 provides indications of focusregions occurring at locations in those images at which focus regionshad not been previously identified, then the processor circuit 350 iscaused to employ various visual recognition algorithms to identify theobjects depicted at those focus regions, and to augment the index 335with indications of what those objects at those focus regions are. Theprocessor circuit 350 is then caused to use this information concerningthese newly identified objects to search for additional images withinthe image database 330 that depict matching or related objects. Theprocessor circuit 350 then operates the interface 390 to transmit asecond subset of the image data 338 a-x that comprises those additionalimages to the viewing device 500 where the additional images can beviewed by the operator of the viewing device 500. Thus, the mere viewingof images of the image database 330, over time and by numerous people,results in the index 335 being improved over time so that desired imagesalready in the image database 330 may be found more readily by others insubsequent searches.

Still further, various pieces of tag information in the index 335 foreach image may be revised over time to adjust weighting values assignedto different pieces of tag information and/or to correct errant taginformation. By way of example, where a search is underway for imagesdepicting a table, one image may be included in a subset of the imagedata 338 a-x that is sent to the viewing device 500 that may actuallydepict a different object somehow misidentified as a table. The eyemovement data 531 may provide an indication of the person viewing theimages on the viewing device 500 viewing the image with themisidentified object in some manner that indicates very different eyemovements from the other images, and this difference may be such that itcalls the identification of that object as a table into doubt (e.g., theperson views the image, but not in a way that leaves a focus regionwhere the misidentified table is). In response, the processor circuit550 may be caused to employ a weighting value on the tag that identifiesthe image as having a table that lowers that tag in prominence, orotherwise indicates there is some doubt as to how useful that tag is.Over time, as other persons seeking to view images of tables havesimilar reactions (as demonstrated by the manner in which they view theimages), the same tag misidentifying the object in that image as a tablemay be so reduced in its weighting that it is finally eliminated fromthe index 335.

As an alternative to the operator beginning their search for an imagewith manual entry and/or selection of text to initially employ a textsearch, the operator may begin by manually selecting and viewing one ormore images from among the images stored within the image database 330.The processor circuit 350 may transmit subsets of the image data 338 a-xthat comprise images grouped by various characteristics that they arefound to have in common as indicated by tags maintained in the index 335(e.g., images captured in the same month or year, images captured by thesame person, images of the same place, etc.) to the viewing device 500.The processor circuit 550 may present these groups of images on thedisplay 580 in random order in two-dimensional grids, or in other waysenabling the operator to quickly select and begin viewing images. Again,as the operator views one or more of the images in these presentedgroups, eye movements of at least one of their eyes are monitored togather eye movement data leading to determinations of fixations,saccades, focus regions, etc. used to find images depicting similar orrelated objects.

It should also be noted that where eye movement data received by theserver 300 (e.g., the eye movement data 131 of the image data 138, orthe eye movement data 531) comprises such “raw” data concerning eyemovements as gaze points, etc., the processor circuit 350 may be causedto augment the index 335 with such “raw” data as part of storingindications of patterns of eye movements in viewing images. It may be(either in addition to or in lieu of other search methods based on focusregions and saccades) that searches for images includes identifyingimages with similar eye movement patterns of persons viewing them. Insuch cases, it may be that objects at the locations of focus regions arenot identified, though the occurrence and locations of focus regions maystill be as part of comparing patterns of eye movement.

In various embodiments, each of the processor circuits 150, 350 and 550may comprise any of a wide variety of commercially available processors,including without limitation, an AMD® Athlon®, Duron® or Opteron®processor; an ARM® application, embedded or secure processor; an IBM®and/or Motorola® DragonBall® or PowerPC® processor; an IBM and/or Sony®Cell processor; or an Intel® Celeron®, Core (2) Duo®, Core (2) Quad®,Core i3®, Core i5®, Core i7®, Atom®, Itanium®, Pentium®, Xeon® orXScale® processor. Further, one or more of these processor circuits maycomprise a multi-core processor (whether the multiple cores coexist onthe same or separate dies), and/or a multi-processor architecture ofsome other variety by which multiple physically separate processors arein some way linked.

In various embodiments, each of the storages 160, 360 and 560 may bebased on any of a wide variety of information storage technologies,possibly including volatile technologies requiring the uninterruptedprovision of electric power, and possibly including technologiesentailing the use of machine-readable storage media that may or may notbe removable. Thus, each of these storages may comprise any of a widevariety of types (or combination of types) of storage device, includingwithout limitation, read-only memory (ROM), random-access memory (RAM),dynamic RAM (DRAM), Double-Data-Rate DRAM (DDR-DRAM), synchronous DRAM(SDRAM), static RAM (SRAM), programmable ROM (PROM), erasableprogrammable ROM (EPROM), electrically erasable programmable ROM(EEPROM), flash memory, polymer memory (e.g., ferroelectric polymermemory), ovonic memory, phase change or ferroelectric memory,silicon-oxide-nitride-oxide-silicon (SONOS) memory, magnetic or opticalcards, one or more individual ferromagnetic disk drives, or a pluralityof storage devices organized into one or more arrays (e.g., multipleferromagnetic disk drives organized into a Redundant Array ofIndependent Disks array, or RAID array). It should be noted thatalthough each of these storages is depicted as a single block, one ormore of these may comprise multiple storage devices that may be based ondiffering storage technologies. Thus, for example, one or more of eachof these depicted storages may represent a combination of an opticaldrive or flash memory card reader by which programs and/or data may bestored and conveyed on some form of machine-readable storage media, aferromagnetic disk drive to store programs and/or data locally for arelatively extended period, and one or more volatile solid state memorydevices enabling relatively quick access to programs and/or data (e.g.,SRAM or DRAM). It should also be noted that each of these storages maybe made up of multiple storage components based on identical storagetechnology, but which may be maintained separately as a result ofspecialization in use (e.g., some DRAM devices employed as a mainstorage while other DRAM devices employed as a distinct frame buffer ofa graphics controller).

In various embodiments, each of the interfaces 190, 390 and 590 employany of a wide variety of signaling technologies enabling each ofcomputing devices 100, 300 and 500 to be coupled through the network 999as has been described. Each of these interfaces comprises circuitryproviding at least some of the requisite functionality to enable suchcoupling. However, each of these interfaces may also be at leastpartially implemented with sequences of instructions executed bycorresponding ones of the processor circuits 150, 350, and 550 (e.g., toimplement a protocol stack or other features). Where one or moreportions of the network 999 employs electrically and/or opticallyconductive cabling, corresponding ones of the interfaces 190, 390 and590 may employ signaling and/or protocols conforming to any of a varietyof industry standards, including without limitation, RS-232C, RS-422,USB, Ethernet (IEEE-802.3) or IEEE-1394. Alternatively or additionally,where one or more portions of the network 999 entails the use ofwireless signal transmission, corresponding ones of the interfaces 190,390 and 590 may employ signaling and/or protocols conforming to any of avariety of industry standards, including without limitation, IEEE802.11a, 802.11b, 802.11g, 802.16, 802.20 (commonly referred to as“Mobile Broadband Wireless Access”); Bluetooth; ZigBee; or a cellularradiotelephone service such as GSM with General Packet Radio Service(GSM/GPRS), CDMA/1xRTT, Enhanced Data Rates for Global Evolution (EDGE),Evolution Data Only/Optimized (EV-DO), Evolution For Data and Voice(EV-DV), High Speed Downlink Packet Access (HSDPA), High Speed UplinkPacket Access (HSUPA), 4G LTE, etc. It should be noted that althougheach of the interfaces 190, 390 and 590 are depicted as a single block,one or more of these may comprise multiple interfaces that may be basedon differing signaling technologies. This may be the case especiallywhere one or more of these interfaces couples corresponding ones of thecomputing devices 100, 300 and 500 to more than one network, eachemploying differing communications technologies.

In various embodiments, each of the controls 120 and 520 may compriseany of a variety of types of manually-operable controls, includingwithout limitation, lever, rocker, pushbutton or other types ofswitches; rotary, sliding or other types of variable controls; touchsensors, proximity sensors, heat sensors or bioelectric sensors, etc.Each of the controls 120 and 520 may comprise manually-operable controlsdisposed upon a casing of corresponding ones of the computing devices100 and 500, and/or may comprise manually-operable controls disposed ona separate casing of a physically separate component of correspondingones of these computing devices (e.g., a remote control coupled to othercomponents via infrared signaling). Alternatively or additionally, thecontrols 120 and 520 may comprise any of a variety of non-tactile userinput components, including without limitation, a microphone by whichsounds may be detected to enable recognition of a verbal command; acamera through which a face or facial expression may be recognized; anaccelerometer by which direction, speed, force, acceleration and/orother characteristics of movement may be detected to enable recognitionof a gesture; etc.

In various embodiments, each of the displays 180 and 580 may be based onany of a variety of display technologies, including without limitation,a liquid crystal display (LCD), including touch-sensitive, color, andthin-film transistor (TFT) LCD; a plasma display; a light emitting diode(LED) display; an organic light emitting diode (OLED) display; a cathoderay tube (CRT) display, three-dimensional projection, augmented reality,etc. Each of these displays may be disposed on a casing of correspondingones of the computing devices 100 and 500, or may be disposed on aseparate casing of a physically separate component of corresponding onesof these computing devices (e.g., a flat panel monitor coupled to othercomponents via cabling).

FIGS. 3, 4 and 5, taken together, illustrate block diagrams of portionsof the block diagram of FIG. 1 depicted in greater detail. Morespecifically, aspects of the operating environments of the capturedevice 100, the server 300 and the viewing device 500 are depicted, inwhich their respective processor circuits 150, 350 and 550 (FIG. 1) arecaused by execution of their respective control routines 140, 340 and540 to perform the aforedescribed functions. As will be recognized bythose skilled in the art, each of the control routines 140, 340 and 540,including the components of which each is composed, are selected to beoperative on whatever type of processor or processors that are selectedto implement each of the processor circuits 150, 350 and 550.

In various embodiments, one or more of the control routines 140, 340 and540 may comprise a combination of an operating system, device driversand/or application-level routines (e.g., so-called “software suites”provided on disc media, “applets” obtained from a remote server, etc.).Where an operating system is included, the operating system may be anyof a variety of available operating systems appropriate for whatevercorresponding ones of the processor circuits 150, 350 and 550, includingwithout limitation, Windows™, OS X™, Linux®, or Android OS™. Where oneor more device drivers are included, those device drivers may providesupport for any of a variety of other components, whether hardware orsoftware components, that comprise one or more of the computing devices100 300 and 500.

Each of the control routines 140, 340 and 540 comprises a communicationscomponent 149, 349 and 549, respectively, executable by correspondingones of the processing circuits 150, 350 and 550 to operatecorresponding ones of the interfaces 190, 390 and 590 to transmit andreceive signals via the network 999 as has been described. As will berecognized by those skilled in the art, each of the communicationscomponents 149, 349 and 549 are selected to be operable with whatevertype of interface technology is selected to implement each of theinterfaces 190, 390 and 590.

Turning more specifically to FIGS. 3 and 5, the control routines 140 and540 comprise a console component 142 and 542, respectively, executableby the processing circuits 150 and 550 to provide user interfaces bywhich images are visually presented on corresponding ones of thedisplays 180 and 580, and possibly by which manual entry or othermechanism of selecting text (or still other mechanism for reception ofinput) is supported via corresponding ones of the controls 120 and 520.Under control of the console component 142, the processor circuit 150 iscaused to visually present on the display 180 an image of what is in theview of the camera 110 as part of providing an electronic equivalent ofa camera eyepiece for an operator of the capture device 100 to use inaiming the camera 110 to capture an image (e.g., the image 880 depictedin FIG. 3). The processor circuit 150 is also caused to monitor thecontrols 120 for an indication of their being operated to cause imagecapture to take place. Following capture and storage of an image as theimage data 138, the processor circuit 150 may be further caused tocontinue to visually present that image on the display 180 for furtherreview by the operator of the capture device 100. Still further, theprocessor circuit 150 may also monitor the controls 120 for indicationsof being operated to manually input or select descriptive text and/orother text associated with the captured image that the processor circuit150 is caused to store as the tag data 132.

Under control of the console component 542, the processor circuit 550 iscaused to visually present on the display 580 images taken from one ormore subsets of the image data 338 a-x received by the viewing device500 from the server 300 as part of enabling the operator of the viewingdevice 500 to search for an image of a particular object of interest tothem. These images may be visually presented on the display 580 in anyof a variety of ways, including individually, in a single-filehorizontal or vertical scrolling fashion, or in a tiled group (possiblyas depicted in FIG. 5) to enable speedy selection for viewing. Further,the processor circuit 550 may also monitor the controls 550 forindications of their being operated to manually input or otherwiseselect search text that the processor circuit 550 is caused to store asthe tag data 532.

The control routines 140 and 540 comprise an eye movement component 141and 541, respectively, executable by the processing circuits 150 and 550to monitor eye movements of an eye (e.g., the eye 888 depicted in FIG.3) gazing at various locations on corresponding ones of the displays 180and 580, using corresponding ones of the eye trackers 181 and 581. Undercontrol of the eye component 141, the processor circuit 150 is caused tooperate the eye tracker 181 to detect eye movement of an eye of anoperator of the capture device 100 during visual presentation of what isin the view of the camera 110 on the display 180 leading up to captureof the image 880, and to then store indications of those eye movementsas the eye movement data 131. Under control of the eye component 541,the processor circuit 550 is caused to operate the eye tracker 581 todetect eye movement of an eye of an operator of the viewing device 500during viewing the visual presentation of images of subsets of the imagedata 338 a-x on the display 580, and to then store indications of thateye movement as the eye movement data 531.

The control routine 140 comprises a capture component 148 executable bythe processor circuit 150 to perform the capturing of an image of whatis in the view of the camera 110, as has been described at length.

Turning more specifically to FIG. 4, the control routine 340 comprisesan eye movement parser 341 executable by the processor circuit 350 toseparate information concerning focus regions from saccades in receivedeye movement data (e.g., the eye movement data 131). The eye parserfurther causes the processor circuit 350 to separately store theindications of focus regions and saccades as the focus region data 333and the saccades data 334, respectively.

The control routine 340 comprises an object identifier 343 and arelationship identifier 344 to examine the focus region data 333 and thesaccades data 334 to derive information associated with an image to beadded to the index 335. The object identifier comprises implementationsone or more visual recognition algorithms to identify objects at thelocations of the focus regions in an image indicated by the fixationdata 333 (whether those objects are simpler objects such as lines andshapes, or more complex objects such as houses and people). Therelationship identifier 344 parses the saccades data 334 to detectpossible relationships between objects that exist in pairs of focusregions linked by high proportions of saccades occurring therebetween(whether directly or through one or more fixation points).

The control routine 340 comprises an index generator 345 executable bythe processor circuit 350 to create entries in the index 335 for newlyadded images, and to augment the index 335 with information concerningobjects identified at locations of focus regions, information concerningpossible relationships between objects suggested by high proportions ofsaccades, and information concerning any tags (textual or otherwise) inreceived tag data (e.g., the tag data 132).

The control routine 340 comprises a search component 347 executable bythe processor circuit 350 to examine the index 335 as part of usingeither received tag data conveying search text (e.g., the tag data 532)or indications of eye movement of a person viewing images provided bythe server 300 (e.g., the eye movement data 531). The search componentincorporates various implementations of text search and other algorithmsto perform searches of the index 335 based on text or other types of taginformation. Where eye movement information is received, the processorcircuit 350 is first caused to examine the index 335 to determine if thefocus regions and saccades identified in the received eye informationare already identified in the index 335. If so, then the searchcomponent 347 causes the processor circuit 350 to search the imagedatabase 330, using the index 335 to identify images depicting objectssimilar to or related to the object(s) identified at the focus regionsidentified in the received eye information. If not, then before such asearch of the image database 330 is performed, the search component 347first passes the location(s) of the newly identified focus regions tothe object identifier 343 to identify objects at those newly identifiedfocus regions and to cause the index 335 to be augmented withinformation concerning those newly identified objects.

FIG. 6 illustrates a block diagram of a variation of the image storageand retrieval system 1000 of FIG. 1. This variation depicted in FIG. 6is similar to what is depicted in FIG. 1 in many ways, and thus, likereference numerals are used to refer to like elements throughout.However, unlike the image storage and retrieval system 1000 depicted inFIG. 1, this variant of the image storage and retrieval system 1000depicted in FIG. 6 includes a variant of the viewing device 500 thatadditionally assumes the database-related functions of the server 300.Thus, the capture device 100 transmits the image data 138 comprising thecaptured image 880 and associated eye movement data 131 and tag data 132via the network 999 (either directly or through another computingdevice, not shown) to the viewing device 500, where the viewing device500 maintains the image database 330. The use of eye movements inorganizing, searching for and refining searches of images issubstantially the same between these two variants of the image storageand retrieval system 1000.

It is envisioned that the image storage and retrieval system 1000depicted in FIG. 1, with its separate and distinct server 300, is likelyto be implemented as part of an image storage and search service on anetwork of an organization working with images, or on the Internet. Insuch a setting, it is envisioned that the server 300 is in thepossession of and operated by a corporate, governmental, or other entitythat is entirely unrelated to what are likely to be the person(s) whopossess and use the capture device 100 and the viewing device 500(possibly the same person). However, it is envisioned that the variantof the image storage and retrieval system 1000 of FIG. 6 might beimplemented solely by one person in possession of both the capturedevice 100 and the viewing device 500, or may be implemented by membersof a family.

FIG. 7 illustrates one embodiment of a logic flow 2100. The logic flow2100 may be representative of some or all of the operations executed byone or more embodiments described herein. More specifically, the logicflow 2100 may illustrate operations performed by the processor circuit150 of the capture device 100 in executing at least the control routine140.

At 2110, a capture device (e.g., the capture device 100) monitors eyemovements of an eye of a person operating it. As has been discussed, theeye movements monitored are of that eye viewing a display of the capturedevice (e.g., the display 180) as that display visually presents animage of whatever is in the view of a camera of the capture device(e.g., the camera 110) to enable the operator to aim the camera atwhatever they select to capture an image of.

At 2120, the capture device awaits a signal indicating operation of itscontrols (e.g., the controls 120) conveying to it a command to capturean image of whatever is in the view of its camera.

At 2130, the capture device captures the image, storing it as imagedata.

At 2132, the capture device stores eye movement data describing eyemovements of the eye of its operator throughout a specified period oftime leading up to the operation of the controls that triggered thecapturing of the image, and possibly afterwards as its operator possiblyviews the now captured image once again. Again, what exactly is storedas data describing eye movements may comprise gaze points detected bythe eye tracker, fixations and/or saccades determined from analyzingwhat the eye tracker detected, and/or focus regions and indications ofhigh proportions of saccades between focus regions or between a focusregion and a fixation.

At 2134, the capture device stores any tag data either manually enteredor otherwise selected by the operator (e.g., descriptive text) orautomatically generated by the capture device (e.g., a timestamp, adate, GPS coordinates, etc.).

At 2140, the capture device transmits the image data, eye movement dataand any tag data to another computing device (e.g., the server 300 orthe variant of the viewing device 500 of FIG. 6).

FIG. 8 illustrates one embodiment of a logic flow 2200. The logic flow2200 may be representative of some or all of the operations executed byone or more embodiments described herein. More specifically, the logicflow 2200 may illustrate operations performed by either of the processorcircuit 350 of the server 300, or the processor circuit 550 of thevariant of the viewing device 500 of FIG. 6.

At 2210, a computing device (e.g., the server 300 or the variant of theviewing device 500 of FIG. 6) receives image data comprising an imagewith eye movement data and/or tag data. Such receipt may be through anetwork shared with numerous devices, or a point-to-point link betweenthe computing device and only one other device.

At 2220, the computing device stores the image in an image database(e.g., the image database 330). At 2230, the computing device augmentsthe index of that image database with whatever tag data was received.

At 2240, the computing device identifies objects located at the focusregions identified by or derived from the received eye movement data,and then augments the index with indications of the identities of thoseobjects at 2242.

At 2250, the computing device determines where high proportions ofsaccades occurred between focus regions identified in or derived fromthe received eye movement data to infer possible relationships amongobjects identified at those focus regions.

At 2252, the computing device augments the index with indications ofthese possible relationships among objects.

FIG. 9 illustrates one embodiment of a logic flow 2300. The logic flow2300 may be representative of some or all of the operations executed byone or more embodiments described herein. More specifically, the logicflow 2300 may illustrate operations performed by either of the processorcircuit 350 of the server 300, or the processor circuit 550 of thevariant of the viewing device 500 of FIG. 6.

At 2310, a computing device (e.g., the server 300 or the variant of theviewing device 500 of FIG. 6) receives search text describing an objectfor which a search of images is to be performed to find images depictingthe object. Such receipt may be through a network or through manuallyoperable controls of the computing device.

At 2320, the computing device searches an index of an image database(e.g., the image database 330) for images described in the index asdepicting the object or other objects related to the object. At 2330,the computing device provides a set of images found in the imagedatabase that are so described. Such provision of images may be througha network or more directly through a display of the computing device.

At 2340, a check is made as to whether any eye movement data has beenreceived providing indications of eye movements of an eye of a personviewing the provided images. If so, then at 2350, the computing devicecompares the focus regions of the received eye movement data to theindex to determine if there are any focus regions indicated in thereceived eye movement data in the provided images that are not alreadyidentified in the index.

If no such new focus regions are identified in the received eye movementdata, then at 2360, the computing device searches the index for imagesdescribed in the index as depicting the objects identified as being atthe focus regions indicated in the received eye movement data, as wellas other objects related to those objects. Then, the computing deviceagain provides a set of images found in the image data to depict suchobjects or such other related objects at 2330.

However, if at 2350, there are such new focus regions, then thecomputing device employs one or more visual recognition algorithms toidentify the objects depicted at those new focus regions at 2352. Thecomputing device then augments the index with indications of theidentities of those newly identified objects at those new focus regionsat 2354, before searching for images described in the index as depictingobjects identified as being depicted at the focus regions indicated inthe received eye movement data, as well as other objects related tothose objects.

FIG. 10 illustrates one embodiment of a logic flow 2400. The logic flow2400 may be representative of some or all of the operations executed byone or more embodiments described herein. More specifically, the logicflow 2400 may illustrate operations performed by either of the processorcircuit 350 of the server 300, or the processor circuit 550 of thevariant of the viewing device 500 of FIG. 6.

At 2410, a computing device (e.g., the server 300 or the variant of theviewing device 500 of FIG. 6) receives search criterion to employ insearching for images other than an object specified as depicted in thoseimages, such as without limitation, a geographic location at which theimages were captured; or a date, an event, a season, etc. during whichthe images were captured. Such receipt may be through a network orthrough manually operable controls of the computing device.

At 2420, the computing device searches an index of an image database(e.g., the image database 330) for images having associated tag data inthe index meeting the search criterion. At 2430, the computing deviceprovides a set of images found in the image database that are sodescribed. Such provision of images may be through a network or moredirectly through a display of the computing device.

At 2440, a check is made as to whether any eye movement data has beenreceived providing indications of eye movements of an eye of a personviewing the provided images. If so, then at 2450, the computing devicecompares the focus regions of the received eye movement data to theindex to determine if there are any focus regions indicated in thereceived eye movement data in the provided images that are not alreadyidentified in the index.

If no such new focus regions are indicated in the received eye movementdata, then at 2460, the computing device searches the index for imagesdescribed in the index as depicting the objects identified as being atthe focus regions indicated in the received eye movement data, as wellas other objects related to those objects. Then, the computing deviceagain provides a set of images found in the image data to depict suchobjects or such other related objects at 2430.

However, if at 2450, there are such new focus regions, then thecomputing device employs one or more visual recognition algorithms toidentify the objects depicted at those new focus regions at 2452. Thecomputing device then augments the index with indications of theidentities of those newly identified objects at those new focus regionsat 2454, before searching for images described in the index as depictingobjects identified as being depicted at the focus regions indicated inthe received eye movement data, as well as other objects related tothose objects.

FIG. 11 illustrates an embodiment of an exemplary processingarchitecture 3100 suitable for implementing various embodiments aspreviously described. More specifically, the processing architecture3100 (or variants thereof) may be implemented as part of one or more ofthe computing devices 100, 300 and 500. It should be noted thatcomponents of the processing architecture 3100 are given referencenumbers in which the last two digits correspond to the last two digitsof reference numbers of components earlier depicted and described aspart of each of the computing devices 100, 300 and 500. This is done asan aid to correlating such components of whichever ones of the computingdevices 100, 300 and 500 may employ this exemplary processingarchitecture in various embodiments.

The processing architecture 3100 includes various elements commonlyemployed in digital processing, including without limitation, one ormore processors, multi-core processors, co-processors, memory units,chipsets, controllers, peripherals, interfaces, oscillators, timingdevices, video cards, audio cards, multimedia input/output (I/O)components, power supplies, etc. As used in this application, the terms“system” and “component” are intended to refer to an entity of acomputing device in which digital processing is carried out, that entitybeing hardware, a combination of hardware and software, software, orsoftware in execution, examples of which are provided by this depictedexemplary processing architecture. For example, a component can be, butis not limited to being, a process running on a processor circuit, theprocessor circuit itself, a storage device (e.g., a hard disk drive,multiple storage drives in an array, etc.) that may employ an opticaland/or magnetic storage medium, an software object, an executablesequence of instructions, a thread of execution, a program, and/or anentire computing device (e.g., an entire computer). By way ofillustration, both an application running on a server and the server canbe a component. One or more components can reside within a processand/or thread of execution, and a component can be localized on onecomputing device and/or distributed between two or more computingdevices. Further, components may be communicatively coupled to eachother by various types of communications media to coordinate operations.The coordination may involve the uni-directional or bi-directionalexchange of information. For instance, the components may communicateinformation in the form of signals communicated over the communicationsmedia. The information can be implemented as signals allocated to one ormore signal lines. Each message may be a signal or a plurality ofsignals transmitted either serially or substantially in parallel.

As depicted, in implementing the processing architecture 3100, acomputing device comprises at least a processor circuit 950, a storage960, an interface 990 to other devices, and coupling 955. As will beexplained, depending on various aspects of a computing deviceimplementing the processing architecture 3100, including its intendeduse and/or conditions of use, such a computing device may furthercomprise additional components, such as without limitation, a displayinterface 985 or a camera 910

The coupling 955 is comprised of one or more buses, point-to-pointinterconnects, transceivers, buffers, crosspoint switches, and/or otherconductors and/or logic that communicatively couples at least theprocessor circuit 950 to the storage 960. The coupling 955 may furthercouple the processor circuit 950 to one or more of the interface 990 andthe display interface 985 (depending on which of these and/or othercomponents are also present). With the processor circuit 950 being socoupled by couplings 955, the processor circuit 950 is able to performthe various ones of the tasks described at length, above, for whicheverones of the computing devices 100, 300 and 500 implement the processingarchitecture 3100. The coupling 955 may be implemented with any of avariety of technologies or combinations of technologies by which signalsare optically and/or electrically conveyed. Further, at least portionsof couplings 955 may employ timings and/or protocols conforming to anyof a wide variety of industry standards, including without limitation,Accelerated Graphics Port (AGP), CardBus, Extended Industry StandardArchitecture (E-ISA), Micro Channel Architecture (MCA), NuBus,Peripheral Component Interconnect (Extended) (PCI-X), PCI Express(PCI-E), Personal Computer Memory Card International Association(PCMCIA) bus, HyperTransport™, QuickPath, and the like.

As previously discussed, the processor circuit 950 (corresponding to oneor more of the processor circuits 150, 350 and 550) may comprise any ofa wide variety of commercially available processors, employing any of awide variety of technologies and implemented with one or more coresphysically combined in any of a number of ways.

As previously discussed, the storage 960 (corresponding to one or moreof the storages 160, 360 and 560) may comprise one or more distinctstorage devices based on any of a wide variety of technologies orcombinations of technologies. More specifically, as depicted, thestorage 960 may comprise one or more of a volatile storage 961 (e.g.,solid state storage based on one or more forms of RAM technology), anon-volatile storage 962 (e.g., solid state, ferromagnetic or otherstorage not requiring a constant provision of electric power to preservetheir contents), and a removable media storage 963 (e.g., removable discor solid state memory card storage by which information may be conveyedbetween computing devices). This depiction of the storage 960 aspossibly comprising multiple distinct types of storage is in recognitionof the commonplace use of more than one type of storage device incomputing devices in which one type provides relatively rapid readingand writing capabilities enabling more rapid manipulation of data by theprocessor circuit 950 (but possibly using a “volatile” technologyconstantly requiring electric power) while another type providesrelatively high density of non-volatile storage (but likely providesrelatively slow reading and writing capabilities).

Given the often different characteristics of different storage devicesemploying different technologies, it is also commonplace for suchdifferent storage devices to be coupled to other portions of a computingdevice through different storage controllers coupled to their differingstorage devices through different interfaces. By way of example, wherethe volatile storage 961 is present and is based on RAM technology, thevolatile storage 961 may be communicatively coupled to coupling 955through a storage controller 965 a providing an appropriate interface tothe volatile storage 961 that perhaps employs row and column addressing,and where the storage controller 965 a may perform row refreshing and/orother maintenance tasks to aid in preserving information stored withinthe volatile storage 961. By way of another example, where thenon-volatile storage 962 is present and comprises one or moreferromagnetic and/or solid-state disk drives, the non-volatile storage962 may be communicatively coupled to coupling 955 through a storagecontroller 965 b providing an appropriate interface to the non-volatilestorage 962 that perhaps employs addressing of blocks of informationand/or of cylinders and sectors. By way of still another example, wherethe removable media storage 963 is present and comprises one or moreoptical and/or solid-state disk drives employing one or more pieces ofremovable machine-readable storage media 969, the removable mediastorage 963 may be communicatively coupled to coupling 955 through astorage controller 965 c providing an appropriate interface to theremovable media storage 963 that perhaps employs addressing of blocks ofinformation, and where the storage controller 965 c may coordinate read,erase and write operations in a manner specific to extending thelifespan of the machine-readable storage media 969.

One or the other of the volatile storage 961 or the non-volatile storage962 may comprise an article of manufacture in the form of amachine-readable storage media on which a routine comprising a sequenceof instructions executable by the processor circuit 950 may be stored,depending on the technologies on which each is based. By way of example,where the non-volatile storage 962 comprises ferromagnetic-based diskdrives (e.g., so-called “hard drives”), each such disk drive typicallyemploys one or more rotating platters on which a coating of magneticallyresponsive particles is deposited and magnetically oriented in variouspatterns to store information, such as a sequence of instructions, in amanner akin to removable storage media such as a floppy diskette. By wayof another example, the non-volatile storage 962 may comprise banks ofsolid-state storage devices to store information, such as sequences ofinstructions, in a manner akin to a compact flash card. Again, it iscommonplace to employ differing types of storage devices in a computingdevice at different times to store executable routines and/or data.Thus, a routine comprising a sequence of instructions to be executed bythe processor circuit 950 may initially be stored on themachine-readable storage media 969, and the removable media storage 963may be subsequently employed in copying that routine to the non-volatilestorage 962 for longer term storage not requiring the continuingpresence of the machine-readable storage media 969 and/or the volatilestorage 961 to enable more rapid access by the processor circuit 950 asthat routine is executed.

As previously discussed, the interface 990 (corresponding to one or moreof the interfaces 190, 390 and 590) may employ any of a variety ofsignaling technologies corresponding to any of a variety ofcommunications technologies that may be employed to communicativelycouple a computing device to one or more other devices. Again, one orboth of various forms of wired or wireless signaling may be employed toenable the processor circuit 950 to interact with input/output devices(e.g., the depicted example keyboard 920 or printer 970) and/or othercomputing devices, possibly through a network (e.g., the network 999) oran interconnected set of networks. In recognition of the often greatlydifferent character of multiple types of signaling and/or protocols thatmust often be supported by any one computing device, the interface 990is depicted as comprising multiple different interface controllers 995a, 995 b and 995 c. The interface controller 995 a may employ any of avariety of types of wired digital serial interface or radio frequencywireless interface to receive serially transmitted messages from userinput devices, such as the depicted keyboard 920 (perhaps correspondingto one or more of the controls 120 and 520). The interface controller995 b may employ any of a variety of cabling-based or wirelesssignaling, timings and/or protocols to access other computing devicesthrough the depicted network 999 (perhaps a network comprising one ormore links, smaller networks, or perhaps the Internet). The interface995 c may employ any of a variety of electrically conductive cablingenabling the use of either serial or parallel signal transmission toconvey data to the depicted printer 970. Other examples of devices thatmay be communicatively coupled through one or more interface controllersof the interface 990 include, without limitation, microphones, remotecontrols, stylus pens, card readers, finger print readers, virtualreality interaction gloves, graphical input tablets, joysticks, otherkeyboards, retina scanners, the touch input component of touch screens,trackballs, various sensors, laser printers, inkjet printers, mechanicalrobots, milling machines, etc.

Where a computing device is communicatively coupled to (or perhaps,actually comprises) a display (e.g., the depicted example display 980,corresponding to one or more of the displays 180 and 580), such acomputing device implementing the processing architecture 3100 may alsocomprise the display interface 985. Although more generalized types ofinterface may be employed in communicatively coupling to a display, thesomewhat specialized additional processing often required in visuallydisplaying various forms of content on a display, as well as thesomewhat specialized nature of the cabling-based interfaces used, oftenmakes the provision of a distinct display interface desirable. Wiredand/or wireless signaling technologies that may be employed by thedisplay interface 985 in a communicative coupling of the display 980 maymake use of signaling and/or protocols that conform to any of a varietyof industry standards, including without limitation, any of a variety ofanalog video interfaces, Digital Video Interface (DVI), DisplayPort,etc.

Further, where the display interface 985 is present in a computingdevice implementing the processing architecture 3100, an eye tracker 981may also be coupled to the interface 985 to track eye movements of atleast one eye of a person viewing the display 980. Alternatively, theeye tracker 981 may be incorporated into the computer architecture 3100in some other manner. The eye tracker 981 may employ any of a variety oftechnologies to monitor eye movements, including and not limited to,infrared light reflection from the cornea.

The camera 910, if present, may employ any of a variety of technologiesto capture images, including a CCD (charge-coupled device) element. Anyof a variety of analog and/or digital interface technologies may beemployed in coupling the camera 910, including various networktechnologies employing any of a variety of visual data transferprotocols.

More generally, the various elements of the devices 100, 300 and 500 maycomprise various hardware elements, software elements, or a combinationof both. Examples of hardware elements may include devices, logicdevices, components, processors, microprocessors, circuits, processorcircuits, circuit elements (e.g., transistors, resistors, capacitors,inductors, and so forth), integrated circuits, application specificintegrated circuits (ASIC), programmable logic devices (PLD), digitalsignal processors (DSP), field programmable gate array (FPGA), memoryunits, logic gates, registers, semiconductor device, chips, microchips,chip sets, and so forth. Examples of software elements may includesoftware components, programs, applications, computer programs,application programs, system programs, software development programs,machine programs, operating system software, middleware, firmware,software modules, routines, subroutines, functions, methods, procedures,software interfaces, application program interfaces (API), instructionsets, computing code, computer code, code segments, computer codesegments, words, values, symbols, or any combination thereof. However,determining whether an embodiment is implemented using hardware elementsand/or software elements may vary in accordance with any number offactors, such as desired computational rate, power levels, heattolerances, processing cycle budget, input data rates, output datarates, memory resources, data bus speeds and other design or performanceconstraints, as desired for a given implementation.

Some embodiments may be described using the expression “one embodiment”or “an embodiment” along with their derivatives. These terms mean that aparticular feature, structure, or characteristic described in connectionwith the embodiment is included in at least one embodiment. Theappearances of the phrase “in one embodiment” in various places in thespecification are not necessarily all referring to the same embodiment.Further, some embodiments may be described using the expression“coupled” and “connected” along with their derivatives. These terms arenot necessarily intended as synonyms for each other. For example, someembodiments may be described using the terms “connected” and/or“coupled” to indicate that two or more elements are in direct physicalor electrical contact with each other. The term “coupled,” however, mayalso mean that two or more elements are not in direct contact with eachother, but yet still co-operate or interact with each other.

It is emphasized that the Abstract of the Disclosure is provided toallow a reader to quickly ascertain the nature of the technicaldisclosure. It is submitted with the understanding that it will not beused to interpret or limit the scope or meaning of the claims. Inaddition, in the foregoing Detailed Description, it can be seen thatvarious features are grouped together in a single embodiment for thepurpose of streamlining the disclosure. This method of disclosure is notto be interpreted as reflecting an intention that the claimedembodiments require more features than are expressly recited in eachclaim. Rather, as the following claims reflect, inventive subject matterlies in less than all features of a single disclosed embodiment. Thusthe following claims are hereby incorporated into the DetailedDescription, with each claim standing on its own as a separateembodiment. In the appended claims, the terms “including” and “in which”are used as the plain-English equivalents of the respective terms“comprising” and “wherein,” respectively. Moreover, the terms “first,”“second,” “third,” and so forth, are used merely as labels, and are notintended to impose numerical requirements on their objects.

What has been described above includes examples of the disclosedarchitecture. It is, of course, not possible to describe everyconceivable combination of components and/or methodologies, but one ofordinary skill in the art may recognize that many further combinationsand permutations are possible. Accordingly, the novel architecture isintended to embrace all such alterations, modifications and variationsthat fall within the spirit and scope of the appended claims. Thedetailed disclosure now turns to providing examples that pertain tofurther embodiments. The examples provided below are not intended to belimiting.

An example of an apparatus comprises a processor circuit and a storagecommunicatively coupled to the processor circuit and arranged to storeinstructions. The instructions are operative on the processor circuit toreceive an image data comprising a first image; store the first image inan image database; receive a first eye movement data associated with thefirst image; determine a first identity of a first object at a firstfocus region in the first image indicated by the first eye movementdata; and augment the image database with an indication of the firstidentity of the first object at the first focus region.

The above example of an apparatus in which the instructions areoperative on the processor circuit to determine a second identity of asecond object at a second focus region in the first image indicated bythe first eye movement data; augment the image database with anindication of the second identity of the second object at the secondfocus region; derive a possible relationship between the first andsecond objects from a plurality of saccades specified by the first eyemovement data as extending between the first and second focus regions;and augment the image database with an indication of the possiblerelationship between the first and second objects.

Either of the above examples of an apparatus in which the image databasecomprising an index, and the instructions operative on the processorcircuit to augment the image database.

Any of the above examples of an apparatus in which the apparatuscomprises an interface operative to communicatively couple the apparatusto a network, and the instructions operative on the processor circuit toreceive the image data from a computing device via the network.

Any of the above examples of an apparatus in which the instructions areoperative on the processor circuit to receive tag data associated withthe first image; and augment the image database with the tag data.

Any of the above examples of an apparatus in which the tag datacomprises one of an indication of a date on which the first image wascaptured, a time of day at which the first image was captured, a placeat which the first image was captured, or an image category of the firstimage is classified.

Any of the above examples of an apparatus in which the instructions areoperative on the processor circuit to receive a search criterion, thesearch criterion comprising one of a defined date of image capture, adefined time of day of image capture, a defined place of image capture,or a defined image category; search the image database for an image thatmeets the criterion; and provide the first image in response to a matchbetween the search criterion and the tag data associated with the firstimage.

Any of the above examples of an apparatus in which the instructions areoperative on the processor circuit to receive a second eye movement dataassociated with the first image; determine a second identity of a secondobject at a second focus region in the first image indicated by thesecond eye movement data; search the image database for an image thatdepicts the second object; and provide a second image that depicts thesecond object from the image database.

Any of the above examples of an apparatus in which the instructions areoperative on the processor circuit to analyze the first image toidentify the image category of the first image and augment the imagedatabase with the image category of the first image.

Any of the above examples of an apparatus in which the tag datacomprises a text description of the first object as depicted in thefirst image, and the instructions operative on the processor circuit toreceive a search text that describes a search object; search the imagedatabase for an image that depicts the search object; and provide thefirst image in response to the tag data that indicates the search objectas depicted in the first image.

Any of the above examples of an apparatus in which the instructions areoperative on the processor circuit to receive a second eye movement dataassociated with the first image; determine a second identity of a secondobject at a second focus region in the first image indicated by thesecond eye movement data; search the image database for an indication ofan image that depicts the second object; and provide a second image thatdepicts the second object from the image database.

Any of the above examples of an apparatus in which the apparatuscomprises a display, and the instructions are operative on the processorcircuit to visually present the second image on the display.

Any of the above examples of an apparatus in which the instructions areoperative on the processor circuit to augment the image database withthe eye movement data, the eye movement data that indicates a pattern ofeye movement from a viewing of the first image and search the imagedatabase for an image indicated as comprising a similar pattern of eyemovement.

An example of another apparatus comprises a processor circuit and astorage communicatively coupled to the processor circuit and arranged tostore instructions. The instructions are operative on the processorcircuit to receive a first eye movement data associated with a firstimage from an image database stored in the storage; determine a firstidentity of a first object at a first focus region in the first imageindicated by the first eye movement data; search the image database foran image that depicts the first object; and provide a second image thatdepicts the first object from the image database.

The above example of another apparatus in which the instructions areoperative on the processor circuit to determine that the image databasecomprises no indication of the first focus region; and augment the imagedatabase with an indication of the first identity of the first object atthe first focus region.

Either of the above examples of another apparatus in which the apparatuscomprises a display, and the instructions are operative on the processorcircuit to visually present the first and second images on the display.

Any of the above examples of another apparatus in which the apparatuscomprises an interface operative to communicatively couple the apparatusto a network, and the instructions are operative on the processorcircuit to receive the first eye movement data from a capture device viathe network.

Any of the above examples of another apparatus in which the instructionsare operative on the processor circuit to receive an image datacomprising the first image; store the first image in the image database;determine the first identity of the first object at a focus region inthe first image indicated by an eye movement data of the image data; andaugment the image database with an indication of the first identity ofthe first object.

Any of the above examples of another apparatus in which the instructionsare operative on the processor circuit to determine a second identity ofa second object at a second focus region in the first image indicated bythe eye movement data of the image data; augment the image database withan indication of the second identity of the second object; derive apossible relationship between the first and second objects from aplurality of saccades specified by the eye movement data of the imagedata; and augment the image database with an indication of the possiblerelationship between the first and second objects.

Any of the above examples of another apparatus in which the instructionsare operative on the processor circuit to search the image database foran image that depicts the second object in response to the indication ofthe possible relationship between the first and second objects andprovide a third image that depicts the second object from the imagedatabase along with the second image.

Any of the above examples of another apparatus in which the apparatuscomprises a display, and the instructions are operative on the processorcircuit to visually present the third image on the display.

An example of a computer-implemented method comprises receiving an imagedata comprising a first image; storing the first image in an imagedatabase; receiving tag data associated with the first image; augmentingthe image database with the tag data; receiving a first eye movementdata associated with the first image; determining a first identity of afirst object at a first focus region in the first image indicated by thefirst eye movement data; and augmenting the image database with anindication of the first identity of the first object at the first focusregion.

The above example of a computer-implemented method in which the methodcomprises determining a second identity of a second object at a secondfocus region in the first image indicated by the first eye movementdata; augmenting the image database with an indication of the secondidentity of the second object at the second focus region; deriving apossible relationship between the first and second objects from aplurality of saccades specified by the first eye movement data asextending between the first and second focus regions; and augmenting theimage database with an indication of the possible relationship betweenthe first and second objects.

Either of the above examples of a computer-implemented method in whichthe image database comprising an index, and augmenting the imagedatabase comprises augmenting the index.

Any of the above examples of a computer-implemented method in which thetag data comprising one of an indication of a date on which the firstimage was captured, a time of day at which the first image was captured,and a place at which the first image was captured.

Any of the above examples of a computer-implemented method in which themethod comprises receiving a search criterion, the search criterioncomprising one of a defined date of image capture, a defined time of dayof image capture, and a defined place of image capture; searching theimage database for an image meeting the criterion; and providing thefirst image in response to the search criterion matching the tag dataassociated with the first image.

Any of the above examples of a computer-implemented method in which themethod comprises receiving a second eye movement data associated withthe first image; determining a second identity of a second object at asecond focus region in the first image indicated by the second eyemovement data; searching the image database for an image depicting thesecond object; and providing a second image depicting the second objectfrom the image database.

Any of the above examples of a computer-implemented method in which thetag data comprising a text description of the first object as depictedin the first image, and the method comprises receiving a search textdescribing a search object; searching the image database for an imagedepicting the search object; and providing the first image in responseto the tag data indicating the search object as depicted in the firstimage.

Any of the above examples of a computer-implemented method in which themethod comprises receiving a second eye movement data associated withthe first image; determining a second identity of a second object at asecond focus region in the first image indicated by the second eyemovement data; searching the image database for an image depicting thesecond object; and providing a second image depicting the second objectfrom the image database.

An example of at least one machine-readable storage medium comprisesinstructions that when executed by a computing device, cause thecomputing device to receive a first eye movement data associated with afirst image provided by the computing device from an image databasestored in a storage of the computing device; determine a first identityof a first object at a first focus region in the first image indicatedby the first eye movement data; search an image database of the imagedatabase for an indication of an image depicting the first object; andprovide a second image depicting the first object from the imagedatabase.

The above example of at least one machine-readable storage medium inwhich the computing device is caused to determine that the imagedatabase comprises no indication of the first focus region and augmentthe image database with an indication of the first identity of the firstobject at the first focus region.

Either of the above examples of the at least one machine-readablestorage medium in which the computing device is caused to receive animage data comprising the first image; store the first image in theimage database; determine the first identity of the first object at afocus region in the first image indicated by an eye movement data of theimage data; and augment the image database with an indication of thefirst identity of the first object.

Any of the above examples of the least one machine-readable storagemedium in which the computing device is caused to determine a secondidentity of a second object at a second focus region in the first imageindicated by the eye movement data of the image data; augment the imagedatabase with an indication of the second identity of the second object;derive a possible relationship between the first and second objects froma plurality of saccades specified by the eye movement data of the imagedata; and augment the image database with an indication of the possiblerelationship between the first and second objects.

Any of the above examples of the at least one machine-readable storagemedium in which the computing device is caused to search the imagedatabase for an image depicting the second object in response to theindication of the possible relationship between the first and secondobjects and provide a third image depicting the second object from theimage database along with the second image.

1-34. (canceled)
 35. An apparatus comprising: an image databaseoperative to store a first image included in received image data, theimage database including an index associated with searching one or moreimages stored in the image database including the first image; aprocessor circuit; a control routine for execution by the processorcircuit to: identify a first focus region in the first image based on afirst eye movement data included in the image data; determine a firstidentity of a first object at the first focus region in the first image;and cause the index to be augmented with an indication of the firstidentity of the first object at the first focus region.
 36. Theapparatus of claim 35, comprising the control routine to: identify asecond focus region in the first image based on the first eye movementdata; determine a second identity of a second object at the second focusregion in the first image; cause the index to be augmented with anindication of the second identity of the second object at the secondfocus region; identify a relationship between the first and secondobjects from a plurality of saccades indicated in the first eye movementdata as extending between the first and second focus regions; and causethe index to be augmented with an indication of the relationship betweenthe first and second objects.
 37. The apparatus of claim 35, comprising:an interface for execution by the processor circuit to communicativelycouple the apparatus to a network and operative to receive the imagedata from a computing device via the network.
 38. The apparatus of claim35, comprising the control routine to: receive tag data associated withthe first image; and cause the index to be augmented with the tag data.39. The apparatus of claim 38, the tag data comprising one of anindication of a date on which the first image was captured, a time ofday at which the first image was captured, GPS coordinates at which thefirst image was captured, a compass direction for a camera that capturedthe first image, or an image category for which the first image isclassified.
 40. The apparatus of claim 39, comprising the controlroutine to: receive a search criterion, the search criterion includingone of a defined date of image capture, a defined time of day of imagecapture, a defined place of image capture that includes GPS coordinates,or a defined image category; use the index to search the image databasefor an image that meets the criterion; and provide the first image inresponse to a match between the search criterion and the tag data usedto augment the index.
 41. The apparatus of claim 40, comprising thecontrol routine to: receive a second eye movement data associated withthe first image; identify a second focus region in the first image basedon the second eye movement data; determine a second identity of a secondobject at the second focus region in the first image; use the index tosearch the image database for an image that depicts the second object;and identify a second image from the image database that includes thesecond object.
 42. The apparatus of claim 41, comprising: the tag dataincluding a weighting value for the image category for which the firstimage is classified; the control routine to analyze the first image toidentify whether the first image matches the image category included inthe tag data; and the control routine to lower the weighting value basedon the first image not matching the image category; and the controlroutine to cause the index to be augmented with the lower weightingvalue for the image category of the first image.
 43. The apparatus ofclaim 39, the tag data comprising a text description of the first objectincluded in the first image, the control routine to: receive a searchtext that describes a search object; use the index to search the imagedatabase for an image that includes the search object; and provide thefirst image in response to the index indicating the search object isincluded in the first image.
 44. The apparatus of claim 43, comprisingthe control routine to: receive a second eye movement data associatedwith the first image; identify a second focus region in the first imagebased on the second eye movement data; determine a second identity of asecond object at the second focus region in the first image; use theindex to search the image database for an indication of an image thatincludes the second object; and provide a second image that includes thesecond object from the image database.
 45. The apparatus of claim 44,comprising: a display communicatively coupled with the control routine;and the control routine to cause the second image to be visuallypresented on the display.
 46. At least one non-transitorymachine-readable storage medium comprising instructions that whenexecuted by a computing device, cause the computing device to: receive afirst eye movement data associated with a first image provided by thecomputing device from an image database maintained at the computingdevice, the image database including an index associated with searchingone or more images stored in the image database; identify a first focusregion in the first image based on the first eye movement data;determine a first identity of a first object at the first focus regionin the first image; use the index to search the image database for anindication of an image including the first object; and provide a secondimage including the first object from the image database.
 47. The atleast one non-transitory machine-readable storage medium of claim 46,the instructions to further cause the computing device to: determinethat the image database includes no indication of the first focusregion; and cause the index to be augmented with an indication of thefirst identity of the first object at the first focus region.
 48. The atleast one non-transitory machine-readable storage medium of claim 46,the instructions to further cause the computing device to: receive animage data comprising the first image; store the first image in theimage database; identify a focus region in the first image based on aneye movement data of the image data; determine the first identity of thefirst object at the focus region in the first image; and cause the indexto be augmented with an indication of the first identity of the firstobject.
 49. The at least one non-transitory machine-readable storagemedium of claim 48, the instructions to further cause the computingdevice to: identify a second focus region in the first image based onthe eye movement data of the image data; determine a second identity ofa second object at the second focus region in the first image; cause theindex to be augmented with an indication of the second identity of thesecond object; identify a relationship between the first and secondobjects from a plurality of saccades indicated by the eye movement dataof the image data; and cause the index to be augmented with anindication of the relationship between the first and second objects. 50.The at least one non-transitory machine-readable storage medium of claim49, the instructions to further cause the computing device to: use theindex to search the image database for an image that includes the secondobject in response to the indication of the relationship between thefirst and second objects; and provide a third image that includes thesecond object from the image database, the third image provided with thesecond image.
 51. A computer-implemented method comprising: receiving animage data comprising a first image; storing the first image in an imagedatabase, the image database including an index associated withsearching one or more images stored in the image database; receiving tagdata associated with the first image; augmenting the index with the tagdata; receiving a first eye movement data associated with the firstimage; identifying a first focus region in the first image based on thefirst eye movement data; determining, by a processor circuit, a firstidentity of a first object at the first focus region in the first image;and augmenting the index with an indication of the first identity of thefirst object at the first focus region.
 52. The computer-implementedmethod of 51, comprising: identifying a second focus region in the firstimage based on the first eye movement data; determining a secondidentity of a second object at the second focus region in the firstimage; augmenting the index with an indication of the second identity ofthe second object at the second focus region; identifying a relationshipbetween the first and second objects from a plurality of saccadesindicated in the first eye movement data as extending between the firstand second focus regions; and augmenting the index with an indication ofthe relationship between the first and second objects.
 53. Thecomputer-implemented method of 51, the tag data comprising one of anindication of a date on which the first image was captured, a time ofday at which the first image was captured, GPS coordinates at which thefirst image was captured, a compass direction for a camera that capturedthe first image, or an image category for which the first image isclassified.
 54. The computer-implemented method of 53, comprising:receiving a search criterion, the search criterion including one of adefined date of image capture, a defined time of day of image capture, adefined place of image capture that includes GPS coordinates, or adefined image category; using the index to search the image database foran image meeting the criterion; and providing the first image inresponse to a match between the search criterion and the tag data usedto augment the index.
 55. The computer-implemented method of 54,comprising: receiving a second eye movement data associated with thefirst image; identifying a second focus region in the first image basedon the second eye movement data; determining a second identity of asecond object at the second focus region in the first image; using theindex to search the image database for an image including the secondobject; and providing a second image including the second object fromthe image database.
 56. The computer-implemented method of 51, the tagdata comprising a text description of the first object as depicted inthe first image, and the method comprising: receiving a search textdescribing a search object; using the index to search the image databasefor an image that includes the search object; and providing the firstimage in response to the index indicating the search object is includedin the first image.
 57. The computer-implemented method of 56,comprising: receiving a second eye movement data associated with thefirst image; identifying a second focus region in the first image basedon the second eye movement data; determining a second identity of asecond object at the second focus region in the first image; using theindex to search the image database for an image that includes the secondobject; and providing a second image including the second object fromthe image database.